AN EXAMPLE OF ORTHOGONAL TRIPLE FLAG VARIETY OF 

FINITE TYPE 



TOSHIHIKO MATSUKI 



Abstract. Let G be the split special orthogonal group of degree 2n + 1 over 
a field F of charF ^ 2. Then we describe G-orbits on the triple flag varieties 
G/PxG/PxG/P and G/PxG/PxG/B with respect to the diagonal action of 
G where P is a maximal parabolic subgroup of G of the shape (n, 1, n) and B is a 
Borel subgroup. As by-products, we also describe GL„-orbits on G/B, Q2ri-orbits 
on the full flag variety of GL2„ where Q2n is the fixed-point subgroup in Sp2„ of 
a nonzero vector in F^" and 1 x Sp2„-orbits on the full fiag variety of GL2„+i. In 
the same way, we can also solve the same problem for S02n where the maximal 
parabolic subgroup P is of the shape (n, n). 



1. Introduction 

Let G be a reductive algebraic group over a field F and let Pi, . . . , be parabolic 
subgroups of G. Then we consider the diagonal action of G on the multiple fiag 
variety 

M = iG/P,)x...xiG/Pk). 
We say Ai is of finite type if it has finite number of G-orbits when the field F is 
infinite. 

In |MWZ99] , Magyar, Weyman and Zelevinsky classified multiple fiag varieties of 
finite type for GL„(F) with an arbitrary algebraically closed field F and described 
their orbit decompositions using quiver theory. In |MWZOO] . they also solved the 
same problem for Sp2„(F). 

Consider a triple fiag variety M = (G/Pi) x (G/P2) x {G/P3) and note that 
G-orbit decompostion on Ai is naturally identified with Pa-orbit decomposition on 
the double fiag variety V = {G/ Pi) x (G/P2). Littelmann ( |L94j ) classified double 
fiag varieties V with open P-orbits for simple algebraic groups G. Here Pi and P2 
are maximal parabolic subgroups of G and B a Borel subgroup of G. Suppose that 
V has an open P-orbit and that F is an algebraically closed field of charF = 0. 
Then it follows from the theorem by Brion ( |B86j ) and Vinberg ( [V86] ) that |P\'E'| 
is finite. 

We can see there are many open problems in this subject. One of them is an 
explicit description of orbit decomposition for each triple fiag variety classified in 
|L94j (Table I). In this paper, we solve this problem for some typical orthogonal 
triple fiag variety. It is interesting that we can describe orbits in our example over 
an arbitrary field of char F 7^ 2 and so we can also compute the number of elements 
in each orbit when F is a finite field. 
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Let F be an arbitrary commutative field of cliarF 7^ 2. Let ( , ) denote tlie 
symmetric bilinear form on F^""*"^ defined by 

for i,j = l,...,2n + l where ei, . . . , e2n+i is the canonical basis of F^""^^. Define the 
special orthogonal group 

G = {ge SL2n+i(F) I {gu,gv) = {u,v) for all u,v e F^^+^j 

with respect to this form. Let us write G = S02n+i(F) in this paper. Let M 
denote the variety consisting of all the maximal isotropic subspaces in F^""*"^. Here 
a subspace V in F^""*"^ is called a maximal isotropic subspace if dimy = n and 
{V,V) = {0}. Then M is a homogeneous space of G and hence it is written as 
M = G/P where P = {g e G \ gUo = Uq} with f/o = Fei © ■ ■ ■ © Fe„ G M is a 
maximal parabolic subgroup of G. 

Let Mo = {Vi G ■■■ C Vn \ {Vn, Vn) = {0}} ^ G/B denote the full flag variety 
of G. Here B is the isotropy subgroup of the canonical full flag Fei C Fei © Fe2 C 
■ ■ ■ C Fei © ■ ■ ■ © Fe„ in Mq which is called a Borel subgroup of G. In this paper, 
we will describe G-orbits on T=MxMxM and % = M x M x Mq with respect 
to the diagonal action. 

In the same way, we can also solve the problem for S02n(F) (Section 1.5). 

Remark 1.1. (i) Similar problems were studied in [KS90j . |FMS04j and [CN06j . 
In particular, |KS90j (p. 492) described Sp2„(M)-orbits on the variety consisting of 
triples of Lagrangian subspaces in a real symplectic vector space. For each orbit in 
this decomposition, there corresponds a symmetric bilinear form and the "Maslov 
index" is naturally defined. So it is natural that there appear alternating forms in 
our results on S02n+i(F)-orbit decompositions of T (Theorem II. 41) and % (Theorem 

OD- 

(ii) We may consider the action of the orthogonal group 

G = 02„+i(F) = {ge GL2„+i(F) | {gu,gv) = {u,v) for all u,v e F^^+^j 

on M and Mq. But since G = G L\ {—g \ g G G} and since — /2n+i acts trivially on 
M and Mq, the G-orbits are the same as the G-orbits. 

(iii) The triple flag variety % has the maximum dimension among the triple flag 
varieties of S02n+i(F) of finite type since 

^ n(n + l) n(n + l) 9 . x , 

dim To = ^ ^ ' + ^ ^ ' +11^ = n{2n + 1) = dim S02n+i(F). 

1.1. G-orbits on T = M x M x M. For d = 0, . . . , n, define f/^ = Fei © • • ■ © 

Fe„_d © Fe„+2 © ■ ■ ■ © Fe„+rf+i G M. For a partition n = a + 6 + c+ + cq + c_ of n 
with nonnegative integers a,6, c+,co and c_, define subspaces 

f/(«) = Fei © ■ ■ ■ © Fe^, U^p) = ¥e2n-a-h+2 © ■ ■ • © Fe2„-a+i, 
f/(+) = Fe„+;,+i © ■ ■ ■ © Fea+6+c+, f/(-) = Fe„+2 © ■ ■ ■ © Fe^+^.+i, 
f/(0) = Fea+6+c+ + l © ■ ■ ■ © ¥ea+b+c++co ® Fen+c_+2 © ■ ■ ■ © Fe„+c_+co + i © IFen+i 
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of]p2n+i_ Write 1^(0) = ^7(a)©^/(/3)©^^(+)©^^(-), k+ = a + b + c+ and k_ = n + c_ + l. 
If Co — 2ci — 1 is odd, then we define an element of M by 



^Cl-l 



Co 



V(a, b, c+, c_)odd = W^(o) ® ( F(efe++i + e^.+O ) © I ^{ek++i - ek_+i) 

\ i=l / \i=ci+l 

© F(ejk_^+ci - 2^k-+ci + Cn+l)- 

If Co = 2ci is even, then we define an element of M by 

(Cl \ / C() 

0F(efc++^ + efe_+i) © F(efc^+, - ek_+i) ] . 

1=1 J \i=ci+l 



If Co = 2ci is even and positive, then we also define 



Cl 



CO — 1 



y(a,&,c+,cXen = W^{o)® 0F(e, ++i + ^k-+i) 1 © I F(efe_^+i — ek_+i) 



=ci+l 



Theorem 1.2. Let t = (V(i), V(2), V(3)) 6e an element ofT = M x M x M. Define 

a = a{t) = dim(y(i) n 1^(2) n V^^)), b = = dim(V^(i) n 1/(2)) - a, 
c+ = c+(i) = dim(V(i) n V(3)) - a, c_ = c_(i) = dim(V'(2) fl V(3)) - a, 
Co = Co(t) = n — a — b — c+ — 
and e = = dim(V'(i) + V^2) + V(3)) + dim(V'(i) n V^2) n V(3)) - 2n e {0, 1}. 

(i) If Cq is odd, then e — 1 and t e G{Uo, Un-a-b, V{a, b, c+, c_)odd)- 

(ii) If Co = 0, then e = and t e G{Uo, U^-a-b, V{a, b, c+, c^)"^^^)- 

(ni) // Co is even and positive, then t e G{Uq, Un-a-b, ^(O; b, c+, c_)g^g^) with 
e = or 1. 



coronary 1.3. |G\r| ^ g (" " 3^ + .Here V. = 1^ HZ'^;^^^ ' 
For n = 1, 2, 3, 4, the number of orbits \G\T\ is as follows. 



n 


1 


2 


3 


4 


\G\r\ 


5 


16 


39 


81 



Theorem 1.4. WhenF is the finite fieldFj. withr elements, the number of elements 
in the G-orbit Gt is 



\Gt\ = |M 



^{n—a){n-a+l)/2 j^'^ 

[r]a[r]b[r]c+[r]c-[r]co 
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Here [r]m is the r -factorial number (r + l)(r^ + r + 1) ■ ■ ■ {r^~^ _|_ . . . _|_ j^-j ^^^^^ 

^Ur) = i^lu-Ar) = = r'^'-'\r - l){r' - I) ■ ■ ■ {r^^-^ - 1). 

Remark 1.5. (c.f. Proposition ll.7p V'co(^) ~ |GLco(F,.)/if^^| where 

{1 X Spcjj_]^(Fr) if Co is odd, 

Spgp(F,.) if Co is even and e = 0, 

Qco = {9 ^ Sp^jj(F,.) \ gv = v} if Co is even and e = 1. 

[v is a nonzero element in .) 

1.2. G-orbits on% = MxMxMq. By TheoremOl we may fix a t = ([/q, t/^, V) 
with G M of the form 

V = V{a, b, c+, c_)odd, V{a, b, c+, c_)°^^j, or V{a, b, c+, C-)^^^^ 

where d = n — a — b. Let Mo(\^) denote the subvariety of Mq consisting of full flags 
J-" : Vi C ■ ■ ■ C Vn satisfying Vn = V. Let vr : To — ?■ T be the projection. Then the 
fiber 7r^^(t) at t is naturally identified with Mo{V). Since the isotropy subgroup at 
t is R{t) = PnPu^n Py. We have only to describe i?(t)-orbits on Mo{V). 

Definition 1.6. A full flag J-" : Vi C ■ ■ ■ C \4 in Mo{V) is called standard if 

V. = {Vi n © {Vi n © {V. n ([/(+) © f/(_))) © {v, n f/(o)), 

V^i n f/(„) = Fei © ■ ■ ■ © Fea^(^) and Vi n [/(;3) = Fe2„-a-6+2 © ■ ■ ■ © Fe2„_a-f,+i+fe,(^) 
for alH = 1, . . . , n where ai(J-') = dim(Vj fl and = dim(Vi fl f/(/3)). 

For a standard full flag J" : 14 C ■ ■ ■ C K, write Ci(J') = dim(\/j fl (f/(+) © f/(-))) 
and di{J^) = dim(l^ fl f/(o))- Deflne subsets 





= {«!, • . 




--{iel 


1 ai{J^) = 


= ai_i(J^) + l} 


A/3) 


= {/3i,.. 


.,M = 




h{:F) = 


6,_i(^) + l}, 


At) 


= {71, • • 


• , 7c} = 




q(-F) = 


Q_i(.F) + l}, 


A<5) 


= {5i,.. 


•,'^co} = 




1 = 


: + 1} 



of / = {1, . . . , ri} where c = c+ + c_ and ai < ■ ■ ■ < c^a, /3i < ■ ■ ■ < /3b, 71 < ■ ■ ■ < 
7c, (5i < ■ ■ ■ < (5co. Let t(J^) denote the permutation 

r( J") : (1 2 • ■ ■ n) (tti ■ ■ ■ a„7i ■ ■ ■ 7^^! ■ ■ ■ ^cqA ■ ■ ■ l^b) 

of / and £(r(J-')) the inversion number of t{J^). 
For X e GL„(F), write 

\ /o 

h[X] = I 1 with J = Jn = 

j'x-^j 
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For A G GLe+(F) 



B e GLco(F) and C G GL^^(F), define an element 



i{A,B,C) = h 



( ^a+b 








A 





0\ 

B 
CJ 



of G. Let L+, Lq, L_, L and Ly denote tlie subgroups of G defined by 



L+ = 
Lo = 
L_ = 

L = X Lq X L_ and Ly - 

Proposition 1.7. (i) Ly = 

(ii) V = V {a, b,c+,c-) odd 

V = V{a,b, c+,c_)°^^j, = 

V = V{a,b, c+,c-)l^^^ = 



CO 5 ^C- , 



{i{A, I, 



AeGU^iW)}, 
I B G GLe„(F)}, 
{£(/e^,4„C) |CgGL,_(F)}, 

= {£ G L I = 1/}, respectively. 

L+ X (Ly n Lq) X L_. 
=^LynLo = lxSp,„_i(F), 
=^LynLo = Sp,„(F), 



Here Qcq = {(? G Sp^jj(F) | (7t> = t>} with some v G F'^" — {0}. 

Theorem 1.8. (i) For every full flag T in Mo(V^), there exists a g E R{t) = 
P n Pu^ n Py such that gj-' is standard. 

(ii) Let T and T' he two standard full flags such that gT = T' for some g G R{t). 
Then there exists a g^ E Ly such that g^J-" = T' . 

(iii) //F = F^, then \R{t)T\ = [r] Jr]fer^(^(^))|LyJ^| for each standard full flag T 
m Mo{V). 

1.3. Orbits on GL„(F)/i?. By Proposition 11.71 and Theorem 11.8^ our problem is 
reduced to the orbit decompositions on the full flag variety of GL„(F) with respect 
to the following four kinds of subgroups H of GL„(F): 



(A) H = GL„^(F) X GL^_ (F) where m+ + m_ = n, 

(B) H = Sp„(F) for even n, 

(C) H = Qn for even n, 

(D) H = lx Sp„_i(F) for odd n. 



When F = C, the subgroups H in (A) and (B) are symmetric subgroups of GL„(C) 
and the orbit structures were described in |M79j and |R79j . We also have symbolic 
description of orbits in [MU90j . 

We will solve Problems (B), (C) and (D) in Section 3. We will also give a proof 
for the Problem (A) in the appendix. We don't need the assumption char ¥ ^ 2 for 
these problems. 

We can express GL„^(F) x GL„_ (F)-orbits on M = GL„,(F)/fi by "+-ab- 
symbols". For example, when m+ = m_ = 2, the orbit structure is as follows 
(Fig. 7 in |MU90j l 
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++ 



++ 




+a— a 



a+a- 



aabb 



a— a+ 



-a+a 



aH — a 



abab 



a — ha 



abba 

Fig.l. GL2(F) X GL2(F)\GL4(F)/S 



Notation: For i — 1, ... ,n — 1, we can consider the partial flag variety 
M, = C • • • C V,_i C C • • • C K-i I dimVj = j} 

and the canonical projection pi : M ^ Mi. For two if -orb its 5*1 and 5*2 in M, we 

write 5*2 when Pi{Si) = Pi{S2) and dim 5*1 + 1 = dim 5*2. (Remark: In our 

setting, every orbit in M is defined by linear equations. So we can define "dimension" 
of each orbit over an arbitrary field F. When F = F^, the number \S\ of points in 
a orbit S* is a polynomial of r and dimS* is the degree of the polynomial.) This 
notation will be also used in Problems (B), (C) and (D). 

Problem (B) is solved as follows. Let G — GL2„(F) and define a nondegenerate 
alternating form ( , ) on F^" by 

^g. e •) - I for i = 1, . . . , n, 

^ \ -5i,2n+\-j for ^ = n + 1, . . . , 2n. 

For a subspace V in F^", let V-^ = {v E V \ {v^V) = {0}} denote the orthogonal 
space for V. Define H = {g & G \ {gu,gv) = {u,v) for all m,^ G F^"}. Then 
H is isomorphic to Sp2„(F). Let : Vi C V2 C • • • C V2n-i be a full flag in 
F^". Define dij = dij{J^) = dim{Vi n Vj^) for i, j = 0, . . . , 2n and = Cij(J') = 
dij-i — dij — di^ij-i + di-ij ior i,j — l,...,2n. We prove the following propositions 
in Section 3.2. 
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Proposition 1.9. The matrix {cj,j}i=i'"'2n ^ symmetric permutation matrix such 
that Ci^i = for i = 1, . . . , 2n. 

Let r = r(J-') be the permutation of / = {1, . . . ,2n} corresponding to {cjj} = 
{cjj(J-')}. Then r is expressed as r = (ii ji) ■ ■ ■ {in in) with transpositions (zi ji), . . . , 
{in in)- We may assume 

if < for t = 1, . . . , n and Zi < Z2 < • ■ ■ < ^n- 

(Hence ii = 1.) Define another permutation a by 

a = o{J^) : (1 2 ■ ■ ■2ra) t-> (zijiZ2j2 ■ ■ ■ Wn)- 

Let E{a) denote the inversion number ^{a) = \{{i,j) \ i < j and (T{i) > (T{j)}\. 
(Remark: We can prove £(r) = n + 2i{a).) 

Proposition 1.10. (i) There exists a basis Vi, . . . , V2n o/F^" satisfying (a) and (b): 

(a) Vi = ¥vi®---®¥vifori = l,..., 2n. 

(b) {vi,Vj) = Cij fori <j. 

(ii) If¥ = ¥r, then the number of bases satisfying the properties (a) and (b) in (i) 
ig (^j- — -^yi-^n+eio-) 

Let C2n denote the set of symmetric permutation matrices {Qj } of degree 2n such 
that Cj j = for i = 1, . . . , 2n. By Proposition 11.91 and Proposition I1.10( we have: 

Corollary 1.11. (i) There exists a one-to-one correspondence between C2n CLnd 
Sp2„(F)\GL2„(F)/i?. 

(ii) |Sp2„(F)\GL2„(F)/i?| = {2n - l){2n - 3) ■ ■ ■ 1 = Ml. 

(iii) If¥ = ¥r, then 

l^^l ^ |Sp2n(F.)| ^ (r^-l)(r^-l)---(r^"-l) ^n^-.-.(.m) 



By this result, we can describe Sp2„(F)\GL2n(F)/-B by the "AB-symbols" . For 
n = 2,3, the orbit structure is as in Fig.2 and Fig.3 (Fig.3 and Fig.4 in |MO90j ). 
For example, the symbol ABBA implies the Sp4-orbit of the flag T such that Ci^4 = 
C2,3 = 1- 

Problem (C) is solved as follows. Retain the notations for Problem (B). Define 
a subgroup Q2n = {g e H \ ge2n = e2„} of H = Sp2„(F). Write W = (Fe2„)"^ = 
Fe2 © ■ ■ ■ © Fe2n- Let S denote the subset of / x / defined by S* = S{J^) = {{i,j) \ 
Vi n Vj^i (tW} and define a subset 

So = So{T) = {{t,j) eS\V,nVj^cW and V,., n V^^_, C W} 
of S. Then the following results will be proved in Section 3.3. 
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ABCCBA 



ABBA 



ABAB 
2 



AABB 
Fig.2. Sp4(F)\GL4(F)/S 



2//4 l\\5 



ABCBCA ABCCAB 




ABCBAC ABBCCA ABCACB 

\4 




ABBCAC ABCABC ABACCB 




ABBACC ABACBC AABCCB 

ABABCC AABCBC 



AABBCC 



Fig.3. Sp6(F)\GL6(F)/S 



Proposition 1.12. (i) If G S'o then Cij — 1. 

(ii) We can write Sq = {{xi, yi), {x2, 1/2), ... , {xs, Vs)} with some Xi < X2 < ■ • • < 
Xs and yi < 1/2 < ■ ■ ■ < Us satisfying {xi, . . . , n {yi, ...,ys}^(f). 

Define m — m(J^) = \{{i,j) \ Cij — 1 and {i,j) e S — So}\. 

Proposition 1.13. (i) There exists a basis vi, . . . , V2n of¥^"' satisfying (a), (b) and 
(c): 

(a) Vi = ¥vi ® ■ ■ ■ ®¥vi for i = 1, . . . , 2n. 

(b) {vi,Vj) = Cij fori < j. 

(c) Vi eW fori^ Xi,...,Xs and {v,^^,e2n) = ■■■ = {v.^^,e2n) = 1. 

(ii) //F = ¥r, then the number of bases satisfying the properties (a), (b) and (c) 

m (i) IS (r - l)n-s^n+i{a)-m _ 
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For a subset Ii^a) of I with even number of elements, write 
C{I[A)) = {symmetric permutation matrices {ci,j}i,jei(A) such that Q^j = for i G I{A)}- 
Theorem 1.14. (i) There exists a one-to-one correspondence between 



\_\\_\C{^A)) and Q2n\GUn{^)/B. 



s=l * 



Here the disjoint union * is taken for all the partitions I — I(^a) U /(x) U I(y) such 



that |/| 



(ii) |g2n\GL2„(F)/S| = Yl 

(iii) If¥ = ¥r, then 

\Q2nJ^\ — 



(2n)! 



^ 2'*-^(s!)2(n-s)!' 

\Q2n\ 



(r^-l)(r^-l)...(r2"-^-l) , _ 



YAAX 




AYAX AYXA YAXA 




AAYX YYXX AXYA 



YXAA 



AAXY YXYX AXAY XAYA XAAY XYAA 

4 
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For n = 1, 2, 3, 4, the number of orbits |(52n\GL2n(F)/i?| is as follows. 



n 


1 


2 


3 


4 


\Q2n\GUn{¥)/B\ 


2 


18 


200 


2730 



We can express orbits by "ABXY-symbols" . When n = 2, the orbit structure is 
as in Fig.4. In the diagram, the symbol XYXY implies the (54-orbit containing the 
flag 7" such that So{T) = {(1, 2), (3, 4)}, for example. 

Finally we will solve Problem (D) only restricting Problem (C) to a subgroup of 
GL2n(IF). Retain the notations in Problem (C). Define a subspace W = Fei © • • • © 
Fe2n-i = (Fei)^ of F^". Then Q'^^ = {g e Q2n I gW = W'} is written as 

QL ^{geH\gei^ei and ge2n = e2n} = 1 x Sp2„_2(F) C GL{W'). 

Consider the variety M' consisting of full flags Vi C ■ ■ ■ C V2„_i satisfying V2n-i = 
W. Then M' is the full flag variety of GL{W'). Two flags in M' are in the same 
Q2n"0^t)it if and only if they are in the same (52n-orbit. 

For a full flag Vi C • • • C V2n-i in M', let i be the least integer such that Vi D Fei. 
Then the pair (i, 2n) is contained in So since Vi fl V2^_i = Fei W. This implies 
Xs — i and ys — 2n. Thus we have: 

Theorem 1.15. (i) There exists a one-to-one correspondence between 

n 

□ □C(V)) and 1 X Sp2„_2(F)\GL2„-i(F)/S. 

s=l * 

Here the disjoint union * is taken for all the partitions I ~ {l,...,2n — 1} = 



(X) U I{Y) such that \I, 



s and that \L 



iy)\ 



s - 1. 



(ii) 11 X Sp,„.,(F)\GL.„_.(F)/5| = J: ,„..^,g'ri)!il _ ,)r 



(iii) 7/F = F^, then 
|(lxSp2„_2(F)).F| = 



|Sp2n-2(F) 



V — \ \n-s{T)^n-\-i{c{T))-m{T) 



(r2-l)(r4-l) 



„2n-2 



-l).(n 



{n-\Y-n-l{p{T))+m(T) 



Here the invariants cr{J^) and m{J-') are defined for the natural extension of the full 
flag T to F^" as is explained above. 



For n < 5, the number of orbits |1 x Sp2„_2(F)\GL2„_i(F)/i?| is as follows. 



n 


1 


2 


3 


4 


5 


|lxSp2„_2(F)\GL2„-i(F)/i?| 


1 


6 


55 


665 


9891 



We can express 1 x Sp2(F)-orbits on GL3(F)/i? as follows: 
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AAX AXA XAA 




XYX 

Fig.5. 1 X Sp2(F)\GL3(F)/5 

We have only to extract from the diagram of Q4\GL4{¥)/B six symbols containing 
the letter "Y" as the fourth letter and then delete these "Y". Clearly the orbit 
structure is the same as GL2(F) x GLi(F)\GL3(F)/5 (c.f. |M()9n] . Fig.5). 

I. 4. Expression by symbols and number of orbits. By Theorem II. 8 [ Corollary 

II. m Theorem 11.141 Theorem 11.151 and Proposition 14. 2[ we can attach each G-orbit 
Gt {t = {Uo,Ud, J^), d = n - a - b) on To = M X M X Mq a "word" w = £i ■ ■ ■ 
consisting of letters ii as follows. Write / = {1, . . . , n} = /(a) U /(^) U J(^) U I(^s) as in 
Section 1.2. Then 

i e I{a) =^ ii = a, i e /(/3) =^ ii = (3, i e I(.y) =^ = +, - or a, b, . . . , 
i e =^ £i = X,Y or A,B, .... 

Here the subword W(^) = ■ ■ ■ expresses an L+ x L_-orbit of the full flag V^-^ fl 
W° C ■ ■ ■ C V^^ nW^ in = f/(+) © f/(_) as in Section 1.3 and Section 4. On the 
other hand, the subword W(^s) = hi ■ ■ ■ expresses an Ly fl Lo-orbit of the full flag 
n t/(o) C ■ • • C Vs^^ n f/(o) in f/(o) as in Section 1.3. 

2 

Example 1.16. (The case of = 2) We can describe G\7^ ^ |J Rd\Mo as in Fig. 6 
when n = 2. 

Notation: Let Pi : Mq — )■ Mj {i = 1, 2) be the canonical projections where Mi = 
{V2 I dimV2 = 2} (= M) and M2 = {Vi \ dimVi = 1} are partial flag varieties. 

Then for two -R^-orbits Si and 5*2 on Mq, we write S2 when Pi{Si) = Pi{S2) 

and dim Si + 1 = dim 5*2. 

Remark 1.17. Suppose F = C. Then G = S05(C) = Sp4(C)/Z2 in this case. So 
the orbit structure is the same as the symplectic triple flag variety of the shape 
(121)(121)(1^) given in IMWZOOj . 

Using these symbols, we can easily count the number of orbits as follows. Let 
^{k) denote the number of words consisting of k letters which do not contain the 
four letters a, /3,+ and — . 



12 
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OLO. 



a/3 



Pa 



1/ 2' 



a- 



++ 

2' 



-\-a 




X^ 



/3X 



-a 



+- -+ 
+X aa AA -X 



X+ 




YX 



XY 



1 



X- 





d = 



d = 1 

2 

Fig.6. □ i?AS05(F)/5 



d = 2 



(i=0 



Lemma 1.18. ^{2k) = V , ..f^f^' and e(2A; - 1) = V 
Theorem 1.19. (i) |G'\7^| = X]4"-^Qc(fc)- 



fc=0 

n 



(ii) |GMF)\Mo| = ^2"-(j^^)e(^). 



fc=0 



For n = 1,2, 3, 4, the number of orbits is as follows. 



// 


1 


2 


3 


1 


ICXTol 


5 


28 


169 


1082 


|GL„(F)\Mo| 


3 


12 


53 


258 



1.5. The case of S02n(F). Let G' and G' be the subgroups of G defined by 
G' ^{g eG\ gcn+i = e„+i} and G' ^ {g e G \ ge^+i = ±e„+i}, 

respectively. Then they are isomorphic to the split special orthogonal group and 
the split orthogonal group of degree 2n, respectively. Let M' be the subvariety of 
M defined hj M' = {V ^ M \ {V, e„+i) = {0}}. 
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Then M' is a homogeneous space of G' consisting of two G"-orbits JVP = G'Uq 
and = G'Ui (f/i = Fei © ■ ■ ■ © Fe„_i © Fe^+s)- Note that V and V in M' 
are contained in the same G'-orbit if and only if n — dim(y fl V) is even. Define 
a subvariety = {J^j C • • ■ C 14 | K e M'} of Mq. Then Mq is also a 
homogeneous space of G' consisting of two G'-orbits Mq and Mq. 

Theorem 1.20. Let t = (V(i), V(2), V(3)) be an element of V = M' x M' x M' . 

Define 

a = a(t) = dim(l^(i) n 1/(2) n 1/(3)), b = h{t) = dim(1/(i) n 1/(2)) - a, 
c+ = c+(t) = dim(1/(i) n 1/(3)) - a, c_ = c_(t) = dim(1/(2) H 1/(3)) - a, 
Co = Co (t) = n — a — 6 — c+ — c_ 
and £ = e{t) = dim(1/(i) + 1^(2) + V(3)) + dim(1/(i) n 1^(2) n 1/(3)) - 2n e {0, 1}. 

Then we have: _ 
(i) e = 0, Co 2S even and t e G'{Uo, Un-a-b, V{a, b, c+, c_)^^^J. 

^,,r("-")("-"-i)[r]„?/'°/(r) 



(ii) If¥ = ¥r, then \G't\ = 2\G't\ = |M'|- 



[r\a[r\b[r\c+ Hcohc- 



[n/2] 



Corollary 1.21. (,) |g'\r| = ^ = g /" - 2«= + 3 



fc=0 



3 

[(n-3)/2] 



= Z^2 = Z^3- 



(ii) |G"\M"i X M'^^ X M^^^l = ^ ( ) + 

[(n-l)/2] /, o\ [("-2)/2] /> , o\ 

(iii) |G'\M- X M- X M-l = ( 3 J + S ( 3 J ^ ^' 



some i, j = 1, 2, 3. 



For n = 2, 3, 4, 5, the number of orbits is as follows. 



n 


2 


3 


4 


5 


\G'\r\ = \G'\r'\/2 


11 


24 


46 


80 


|G"\M^i X M^2 X M^3| (^y^ = z/2 = z/3) 


5 


6 


16 


20 


|G'\M^i X M'^^ X W-'l {ui ^ Uj for some i, j) 


2 


6 


10 


20 



Let t = {Uo, Udi V) with d = n — a — b and V = V{a, b, c+, c_ 
proposition is proved in the same way as Theorem 11.81 



/ even 



The following 



Proposition 1.22. (i) For every full flag J-" in MoiV) fl Mq, there exists a g E 
R{t) n G" such that gj^ is standard. 

(ii) Let T and T' be two standard full flags in Mq(1/) fl Mq. Then T' = gj-" for 
some g e R{t) fl G' =^ T' = giJ^ for some g^ G Ly fl G' . 

(iii) LetF be a standard full flag inMo{V)nM[^. If¥ = ¥,., then \{R{t)nG')J^\ = 
[rl[r]yiri^))\{LynG')T\. 
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As in Section 1.4, we can express G'-orbits on 7^' = M' x M' x Mq by words with 
letters a, /3, +, — , a, b,... and A, B, . . .. The following corollary is proved in the 
same way as Theorem 11.191 

W2] [n/2] ,„_2fc. 



\r"\'T'\ ^ ' ' / \ ('9i-M 
Corollary 1.23. (i) |G'\7;'| = = J] 4'-(;j M: = J] 

I.— n ^ / I-— n 



... k\(n-2ky: 



\Ct'\T'\ -n' 

(ii) |G'\M^i X X Mg^^l = ' ] °' + /X • w;/iere 

ifnis odd, 
H = ■{ 1 ifnis even and Ui = 1^2, 
-1 if n is even and Vi ^ V2- 

(iii) |GL„(F)\M^| = 2|GL„(F)\M°| = ^ ^ ^ ■ 
For n = 2, 3, 4, 5, the number of orbits is as follows. 



n 


2 


3 


4 


5 




\G'\Tl\ = \G'\T^\/2 
G'XM"^ X M"^ X M^^ 1 (z/i = z/2) 
G'\M^i X M-^^ X M^' 1 (z/i ^ z/2) 
|GL„(F)\M^| =2|GL„(F)\M0| 


18 
5 
4 
6 


88 
22 
22 
20 


460 
118 
112 
76 


2544 
636 
636 
312 



Remark 1.24. When F = C, GL„(C) is a symmetric subgroup of S02n(C). The 
structure of GL„(C)\M0 ^ GL„(C) \S02n(C)/E is described in |MU90] (Fig. 19 and 
Fig.20) for n = 3 and 4. (In |MU90] . read GL{n,C) for x PSL{n,C)- For A + + 
in Fig.20 read AA+.) 



2. Orbits on M and Mq 

2.1. Preliminaries. First we prepare some results on orbits on M and Mq which 
follow essentially from the Bruhat decompositions for the Chevalley-type groups. 
Since we need more explicit results, we will prove them by elementary arguments. 

Write Wo = ¥ei®-- ■ ©Fe„„rf, Wi = Fe^^d+i © ■ ■ • ©Fe„+d+i and W2 = Fe„+rf+2 © 
■ ■ ■ © Fe2n+i- Then = Wq © Wi. Define a maximal parabolic subgroup = 
{g E G \ gWo = Wq} of G. Let N^^, Ly/^ and Lvki be subgroups of defined by 



'A 

Nw, = {\ hd+i * \ }, Lwo = < \ hd+i 





* 


* 







* 














and Lwi = {(7 G G | (^f = f for all w G Wq 




A e GL™(F) 
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respectively, where m = n — d. Then is the unipotent radical of Pwo and 
Lw^Lw^ = X is a Levi subgroup of Pwo- The subgroup Lw^ is identified 
with the special orthogonal group for W^. Define 

Write 



where X — {xij} is a {2d+l) x m matrix and Z — {zij} is an m x m matrix. Then 
we can write 

(2.1) Nw, = {g{X,Z) \Z+Jm'ZJm = -Jm'XJ2d+lX}. 

Noting that 

2d+l 

-k,j 




2 

k=l 



2d+\ 



(2.2) Zm+l-j,j — ~^ Xk,jX2d+2- 

ior j — 1, . . . ,171 and 

(2-3) Zij — —Zfn+l-j,m+l-i ~ ^ ] ^k,j^2d+2-k,m+l-i 

k=l 

for i + J > 171 + 2, we have: 

Lemma 2.1. There exists a bijection between W'^^'^'^'^^^'^'^^"^'^^/'^ and Nw^ given by 

(X, {zi,j]i+j<m) ^ g{X, Z). 



Definition 2.2. A maximally isotropic subspace V in F^"+^ is called standard if 

V ^ {V f\WQ) ® {V f\ Wi) e (y n W2), 

VnW^^ C/(a) (= Fei e • • • e Fe„) 
and VnW2^ %) (= Fe„+d+2 • • • Fe2„-a+i) 
with some a = 0, . . . , m = n — d. 

Proposition 2.3. (i) For every V e M, t/iere ea;isfe a g & Q such that gV is 
standard. 

(ii) Let V and V be two standard elements in M. If V — gV for some g — 
gqgL e Pwo = QLw^, then V = giV . 

(iii) When F = F,, IQ^I = j.^{n-a){n-a+i)-d{d+i))/2 ^^a+b y ^ ^ Here a = 

[na[r]b 

dim(yniyo), b = n-d-a and[r]k = (r + l)(r^ + r + 1) • • • (r*^"^ + r*^-^ H hi). 
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Proof, (i) Let tt2 : F^"+^ — )■ W2 denote the projection with respect to the direct sum 
decomposition F^^+i = Wq®Wi® W2. By the action of Lwo = GL{Wo), we may 
assume V^fl VFq = K is equivalent to 7r2(V") = ?7(/3). Take vectors Ui, . . . ,Ub & V 
such that Tr2{uj) = Cn+d+i+j for j = 1, . . . ,b. Then we can write 

m 2d+l 

(^n+d+l+j ~l~ ^ ^ ~l~ ^ ^ •^i,j^m+i 

1=1 1=1 

with some Zij, Xij G F. It follows from the condition {uj, Uk) = for j, /c = 1, . . . , 6 
that 

2d+l 

Zm+l-j,j = ~2 ^ ] ^k,jX2d+2-k,j 
k=l 

for j = 1, . . . ,b and that 

j,m+l— « ^ ^ k,j-^2d+2—k,m+l—i 
k=l 

for G \ i+ j>rn + 2, j<b}. Hence we can take X = {xij} and 

Z = {zij} satisfying (12. 2p and (12. 3p so that Xij = Xij and that Zij = Zij for 
j = 1, . . . ,b. Take g = g{X, Z) G N-^^y Then we have 

^"V = (^-V n (H^o © VTi)) © (^-V n Vrs) and g-Wf\W2 = U^py 
Since V n {Wq © IVi) ± ^("V n W2, we have 

^-V n {Wo © VTi) c © VTi. 

Hence 

n {Wo © iVi) = {g-^v n w^o) © (^"V n Wi) 
with n VTo = 1/ n VTo = 

(ii) The condition V = gV implies dim(V^ fl Wq) = dim(V^' n Wq)- So we have 

y. n PFo = K n w^o = f/{a) and v,nW2 = v,'nW2 = U(^^y 

We have only to show that gL{V fl Wi) = V n Wi. Let v be an element of fl Wi. 
Then we have 

gveV'n W^ = {V n Wo) © n Wi). 

So we can write gv = vo + fi with some f G V^' fl Wq and f 1 G V fl W^i. Since 
gLV G and since gv = gqgiv G giv + Wo, it follows that giv = f 1. Thus we have 
proved gL{V fl Wi) = V'nWi since dim(l^ fl Wi) = dim{V' fl l^i). 

(iii) By (i), we may assume that V is standard. Note that we may consider iV 
instead of V with some £ G because iQi~^ = Q. So we may assume 

VnWi = ¥em+i © • ■ ■ © Fe„. 

Let g = g{X, Z) be an element of Nwq such that gV = V. Since g{V fl W2) C V, 
we have 

(2.4) i>d+l, 3 <b^ Xij = 
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and 

(2.5) i>a + l, j <b^ Zij = 0. 

It follows from ([22D, (^M and (ED that = for j = 1, . . . , 6 and that 

i for G \ j <b, i+j>m + 2}. Hence the condition 

(12.51) follows from the condition (12.41) and the condition 

(2.6) z > a + 1, j <b, i+j<a + b^ Zij = 0. 

Conversely suppose that g = g{X, Z) satisfies the conditions (12. 4p and (12. 6p . Then 
g{VnW2) C V. Write 

m 

i=l 

for j = l,...,2d+ 1. Then = -X2d+2-j,m+i-i by (jSII])- Since 

Vij = -X2d+2-j,m+i-i = for E | i > a + 1, j <d+l} 

by ([23D, we also have g{V fl W^i) C V. Hence = V. 
Thus it follows from Lemma [2. II that 

\NwoV\ = \NwM\{9 e Nw, I = ^1 = r'^^+'^+'^'-'y^ 

^ ^((n-a)(n-a+l)-d((i+l))/2 

On the other hand, we have |Lvi/o^| = ['"]a+6/['"]aM6 since L\Yo — GL(Wo) and 
h(y n W2) is the orthogonal subspace of h{V fl Wq) in W2 for /i e Lq. Thus we have 
the desired formula for \QV\. □ 

Fix a standard maximally isotropic subspace V in F^""*"^ and let Mq{V) denote 
the subvariety of Mq consisting of full flags J-" : Vi C ■ ■ • C \4 such that Vn = V. 
For a full flag J-" : Vi C ■ ■ ■ C \4 contained in MoiV), define 

a, = ai{T) = dim{Vi D Wo), Q = c,(J-) = dim{Vi D W^^) - dim{Vi D Wq) 

and bi = bi{F) = dim Vi - dim{Vi n W^) 
for i = 0,1, . . . ,n. Define subsets 

I{a) = {ai,. . ., Oa} = {i e I \ ai = ai_i + 1}, 
^(A) = {Ai, . . . ,\d} = {i e I \ Ci = Q_i + 1}, 
/(;3) = ...,/3b} = {ie I \b, = bi-i + 1} 

of / = {1, . . . , n} with ai < • ■ ■ < aa, Xi < ■ ■ ■ < Xd and /3i < • • ■ < Then 
/ = /(a) U /(A) U /(^). Consider the permutation 

a = a{J^) : (1 2 ■ ■ ■ n) h-^ (tti ■ ■ ■ OaAi ■ ■ ■ Ad/3i ■ ■ ■ /?&) 

of /. Then the inversion number ^{a) of cr is 

%) = |{(z, j) I a, > A,}| + \{{i,3) I a, > + |{(z, j) | A, > 
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Definition 2.4. A full flag J' : Vi C ■ ■ ■ C Vn = V in Mo{V) is called Lvi/i-standard 
if 

Vi = {Vi n Wo) © (Vi n Wi) © {v^ n W2), 

y^niyo = Fei©---©Fe,,(^) 
and V,nW2 = ¥en+d+2 © ■ ■ ■ © Fe^+d+i+b^iT) 

for z = 1, . . . , n. 

Write Qv = Q Ci Py and (Lvi/Jy = H Py. By Proposition 12.31 (ii), we have 

Pwo n = Qv{L]vJv 

Proposition 2.5. (i) For every full flag : Vi G ■ ■ ■ G Vn = V in Mq{V), there 
exists a g E Qv such that gj^ : gVi G ■ ■ ■ G gVn = V is L\y^- standard. 

(ii) Let T and T' he two L^^-standard full flags in Mo(V"). If F' = gF for some 
9 = 9q9l e Pwo n Py = QviLwi)v, then T = giJ^. 

(iii) When F = F^, we have \QvJ^\ = r^^''^^'^^[r]a[r]b- 
Proof, (i) Since V is standard, it is written as 

V = {vnWo)®{vn Wi) ®{vn W2), 

with V n Wq = U(^a) and V fl W2 = U(^j3y We may moreover assume V fl Wi = 
Fcm+i © ■ ■ ■ © Fe„ replacing V by iV with some i G Lwi- By the same reason, we 
may assume 

7ii{Vx, n {Wo © m)) = Fe^+i © ■ ■ ■ © ¥e,n+^ 
for i = 1, . . . ,d where tti : Wq © Wi Wi is the projection. 
By the action of L]y^ H Py, we may assume 

V,nWo = ¥ei®---®¥ea,^T) and 7r2(\/,) = Fe„+d+2 © ■ ■ ■ © Fen+d+i+6,(^) 
for z = 1, . . . , n. We can take vectors 



Wj 



i=l 

for J = 1, . . . , c? such that ¥wi © ■ ■ ■ © ¥wj G Vx^ We can also take vectors 

a d 

W'j = en+d+l+j + "^ZijCi + y^^Xj J Cm+i G V 
1=1 i=l 

for j = 1, . . . , 6 such that ¥w[ © ■ ■ ■ © ¥wj G V^.. Take g = g{X, Z) e Nw^ n Py 
with X = {xij} and Z = {zij} so that 

Xij for i < d, j < b, 

for i > d + 1, J < 6, 

for i < rf + 1, j > 6 + 1, 

^ym+i-j,2d+2-i ioT i>d + 2, j >b+l, 

Zij for i < a, j < b, 
if z > a + 1 or j > 6 + 1. 
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Then we have 

g-^V, = {g-'V, n Wo) © ig~% n W^) © {g-% n W^) 

with g-^Vi n VTo = Fei © • • ■ © Fe„,^(^), g~^Vi nWi= Fe^+i © ■ • • © Fe„+e,(j-) and 
g'^Vi nW2 = Fe„+d+2 © • ■ ■ © ¥en+d+i+h{T) for i = 1, . . . ,n. 

(ii) The condition J^' = gj^ imphes aj(J-') = ai{J-'') and = h^J^') for i = 
1, . . . , n. So we have 

v,r}Wo = VI n Wo and n = K n W2. 

We have only to show that gh^Yi H VTi) = VI fl VTi for i = 1, . . . , n. Let w be an 
element of fl W\. Then we have 

gv e V/ n w^ = {V^ n Wo) © (V/ n H^i). 

So we can write (yff = t>o + Vi with some t>o G V/ fl l^o and fi G V^/ fl l^i. Since 
gLV G W^i and since gv = gqghV G (y'L'i^ + Wo, it follows that gLV = f i. Thus we have 
proved giiV H W^i) = 1// H W^i since dim{Vi n W^i) = dim(\// fl Wi). 

(iii) By (i), we may assume that J-" is Lvvi- standard. in the proof of (i), we 
may assume Vx^ Pi Wi = Fe^+i © ■ ■ ■ © Fe^+i for i = 1, . . . , Note that 

gV = V ^ dMD and f l23|) 

for (7 = ^?(X,Z) GiVw/o- 

Suppose 5^ J-" = J-". Then since gCm+j G Vx^. , we have 

i < a, ai > Xj =^ = —X2d+2-j,m+i-i = 

for j = 1, . . . ,d. Since gcn+d+i+j G Vp^, we also have 

i < d, Xi > f3j =^ Xij = 

and i < a, ai > (3j =^ Zij = 

for J = 1, . . . ,b. Conversely if (? G Nwo H Py satisfies the above three conditions, 
then gj^ = T . 
Thus we have 

n Py)T\ = \Nwo n Pv\/\{g e Nw, r\Pv\gJ' = J'}\ = r'^''^^^\ 

Clearly \{Lwo H Pv)^\ = MaMb (the product of the numbers of full flags in fl Wo 
and VnW2). So we have \QvJ^\ = r^('^(^))[r]„[r]6. □ 



2.2. First reduction. First apply Proposition 12.31 to the case of = 0. Noting 
that Q = Pwo = P in this case, we get the P-orbit decomposition 

n 

(2.7) M = [_\PU, 

of M. (Of course, we can also deduce this from the Bruhat decomposition of G.) 
Furthermore if F = F^, then 

(2.8) \PU,\ = r*(^+^)/2. 
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by Proposition [23] (iii). By (12. 7p . we have a decomposition MxM = \_\"=Q{{gUo, gUi) 
g G G}. Since the isotropy subgroup of G at ([/q, Ui) eMxM is Ri = Pr\ Pu^, we 
can write 

n 

(2.9) M X M ^\_\G/Ri. 

By (12. 9p . we have only to describe G-orbits on (G/Ri) x M and (G/Ri) x Mq with 
respect to the diagonal action of G for i = 0, . . . ,n. These orbit decompositions are 
identified with the i?j-orbit decompositions of M and Mq, respectively. 

2.3. Second reduction. Consider i^^^-orbit decompositions of M and Mq for d = 
0, . . . ,n. Since Uq nUd = Fei © ■ ■ ■ © Fe„_d = Wq, we have Q <Z Rd C Pwo in the 
setting of Section 2.1. So we can apply the results in Section 2.1 once more. By 
Proposition 12.31 and Proposition 12. 5[ the problems are reduced to the fl-R^-orbit 
decompositions of M and Mq. Note that 

Lw, nRd = {gE Lw, I g{Wi n f/o) = WinU^ and g{Wi n Ud) = H^i n Ud] = GLd(F) 

since {Wi fl f/o) H (W^i H f/^) = {0}. So we will consider the case of d = n in the next 
subsection. 



2.4. The case of = ra. Assume d = n. For A G GL„,(F), define 

/A \ /or 
=01 with j = j:„ = 

Vo Vi 0, 

as in Section 1.1. Then we can write Rn = {h[A] \ A E GL„(F)} = GL„(F). Write 
H = Rn in this subsection. For a maximal isotropic subspace V in F^""*"^, define 

c+ = c+{V) = dim{V n Uq), c„ = c_(F) = dim{V fl f/„), Cq = Co{V) = n - c+ - 

and e(V) = dim{V + (f/o © f/„)) - 2n G {0, 1}. 
Let M(f/(_(_), f^(-)) denote the subvariety of M consisting of V E M such that 

V n f/o = f/(+) = Fei © ■ ■ ■ ©Fec+ and that V n f/„ = f/(_) = Fe„+2 © ■ ■ ■ ©Fe„+c_+i. 

By the action of iif = GL„(F), we may assume V G M(f/(+), f/(_)). Let 7r+ : 
p2n+i _^ ^_ . p2n+i ^ g^^^ . p2n+i ^ ^ = ^g^^^ deuotc the projectlous 

with respect to the direct sum decomposition F^*^"^^ = f/o © f/„ © Z. Write f/(o+) = 
Fec++i © ■ ■ ■ © Fec++co and f/(o-) = Fe„+c_+2 © ■ ■ ■ © Fcn+c.+co+i- Then 

f/(o) = t^(o+) © f/(o-) © ^• 

Lemma 2.6. (i) ker7r+|y = f/(_) anc? ker7r_|y = f/(+). 

(ii) Tx+{V) = f/(+) © f/(o+) and 7i-{V) = f/(_) © f/(o_). 

(iii) r/ie inner product { , ) is nondegenerate on the pair (f/(o+), f/^(o_)). 
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Proof, (i) By the symmetry, we have only to prove the first equahty. The kernel of 
the map ii+\v '■ V —> Uq is Vfl (f/„ ©Z). If an element w G is written as v = + z 
with v_ G Un and z & Z, then we have 

= (f , v) = (f _ + z,v^ + z) = {z, z) 

and hence z = 0. Thus we have V r\{Un® Z) = V r\Un = f/(-). 

(ii) For u & V and f G f/(_), we have 

= {U,V) = {7l+{u) + 7T^{u) + 7fz{u),v) = {7T+{u),v). 

Hence tt+{V) C © f^(o+)- On the other hand, the dimension of n+iV) is n — 
dimf/(_) = n — c_ by (i). Hence the equality holds. By the symmetry, we have the 
second formula. 

(iii) is clear from the definition. □ 
Corollary 2.7. V = © f/(_) © (V n f/(o)). 

By Lemma [2.61 (i) and (ii), we have linear isomorphisms 

Ti+lvnu^o) -y^U^o) t/(o+) and 7r_|yni/(o) -VnU^o) [/(o_). 
So we can define a linear isomorphism 

/y = 7r_|ync/(o) ° (^+|vnc/(o))~"^ : f^(o+) t^(o-)- 

We also define a linear form yjy : f^(o+) — > F by 

</'y(f) = (en+i,7r+|^^^^^j(t;)). 

By Lemma (iii) and the above argument, we can define a nondegenerate bilinear 
form on ?7(o+) by 

{u,v)v = ifviu),v) 

for M,f G ?/(o+)- Define the alternating part and the symmetric part of ( , )y by 

{u, w)y = ^((m, v)v - {v, u)v) and (m, f )'^°' = ^(("' + 

respectively. Let m = + + and v = + vz + v_ be elements of fl U{q) 
with G f/(o+)) Uz,vz G Z and G f^(o-)- Then we have 

= (m, f ) = {u+ + uz + t;+ + -u^ + -u-) 

= {u+,v.) + {uz,vz) + {u-,v+) 

= iu+, fviv+)) + (uz, vz) + ifviu+), v+). 

Hence we have 

(2.10) (m,^^)^" = ^{{u,v)v + {v,u)v) = -^ipv{u)>fv{v) 
for M,f G ?/(o+)- In particular, if m or f is in the kernel of ipv, then 

(2.11) {u,v)v= {u,v)f. 

Lemma 2.8. (i) If Co{V) = 0, then e{V) = 0. 
(ii) Ifco{V) IS odd, then e{V) = 1. 
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Proof, (i) If Co{V) = 0, then V = [/(+) © f/(_). Hence V C Uo ® U„. 

(ii) If 6{V) = 0, then the bilinear form ( , )v is alternating by ( 12. lip . Since it is 
also nondegenerate, the dimension Co{V) = dimf/(o+) is even. □ 

Proposition 2.9. (i) If Cq is even, then ( , is nondegenerate on f/{o+)- 
(ii) If Co is odd, then ( , )y* is nondegenerate on y9y^(0). 

Proof (i) By (12. lip , we may assume that (pv is nontrivial. Suppose that ( , )y* is 
degenerate. Then the subspace 

Y = {ye f/(o+) I {y, v)^' = for all v G t/(o+)} 

of ?/(o+) is nontrivial and even-dimensional. Take a nonzero element y of F ny9y^(0). 
Then for all v G ?7(o+), it follows from (12. lip that {y,v)v = {y,v)v^ = 0, contradict- 
ing that ( , )i/ is nondegenerate on f/{o+)- 

(ii) By (12. lip . ( , )y = ( , )y* on Lpy^{0). Suppose that it is degenerate on Lpy^(0). 
Then the subspace 

Y = {ye vy\0) I {y,v)v = for all v G ^y\0)} 

of <y2y^(0) is nontrivial and even- dimensional. Take a vq E ?7(o+) — V^\/^(0) and write 

Y' = {yeY\ {y,vo)v = 0}. 

Then F' is nontrivial and {y,v)v = for all v G ?7(o+), contradicting that ( , )y is 
nondegenerate on f/(o+)- D 

Proposition 2.10. Let V and V be two elements of M{U(^^),U(^)). Then the 
following three conditions are equivalent 

(i) V = v. 

(ii) fv = fv and Lfy = ^pv ■ 

(iii) ( , )f = ( , andipv = W- 

Proof, (ii)^^(i). By the direct sum decomposition f/(o) = f/(o+) ©^©f/(o-)5 Vr\U{Q) 
is written as 

V n f/(0) = {{ll+iy), TTz{v), TT^{v)) \ V e V n U(^o)} = {{U, (Pv{u)en, fv{u)) I U G f/(0+)}- 

Hence fl f/(o) is determined by the two maps fv and ipv- 

(iii)^^(ii). By (I2.10p . ( , )y™ is determined by ipv- Hence the bilinear form 
( , )y is determined by ( , )y* and ipv- Since the inner product ( , ) defines a 
nondegenerate pairing between ?7(o+) and f^(o-)) the map fv is determined by ( , )v- 

The implication (i)^^(iii) is trivial. □ 

Proposition 2.11. (i) Let V be an element of M. Write c± = c±{V), cq = cq{V) 
and 6 = eiy). Then 

{HV{0, 0, c+, c_)odd «/ Co is odd, 
HV{Q, 0, c+, c_)°yp^ if Co is even and e = 0, 
HV{Q, 0, c+, c_)gygn if Co 2S even and e = 1. 



fii) Let Ln 
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GL^„(F) be as in Section 1.2. Then 



{ieLo\iV = V} 



'l X Sp,^,_i(F) tfV = \/(0,0,c+,c_)odd, 
SPeo(F) z/l^ = \/(0,0,c+,c_)L„, 

Qco ifV = V{0,0,c+,c.'' 

where Qcq = {(? G Spc^(F) \ gv = v} with some nonzero v G F^°. 
(iii) If¥ = ¥r, then 

\HV\ = . . ^g.,(r). 



[r\c+[r\ 
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Proof, (i) By the action of H = Rn, we may assume VHUq = [/(+) and VnUn = 

By the above arguments, the space V defines an alternating form ( , on f/(o+) 

and a hnear form (fy '■ f^(o+) ~^ ^■ 

For V = V{0, 0, c+, c_)gygn, the form ( , is standard: 



Si^co+i-j if « < Co/2, 
-5i^co+i-j if ^ > Co/2. 



The hnear form (pv vanishes for V = V{0, 0, c+, c_ 



and </?y(e, 



Si m for 



1/ = 1/(0, 0, c+, c_)^^g„. On the other hand, for V = 1/(0, 0, c+, c_)odd (co = 2ci - 1), 
we have 



■'t,C0-J 





-5. 



if z < Ci, 
if z = Ci, 
*iCo~j if i > Ci , 







^^c, 






I' 


1 


A 





1 






^0 







and (/3v'(ec++j) = Si,ci- Note that ( , )q is nondegenerate oiaW = ^Py^iO) = Fcc^+i 
• • • © Fec++ci-i © Fe,++,,+i © ■ ■ ■ © Fcc^+co- 
Let Lo be the subgroup of H defined by 



A e GL,„(F) } = GL([/(o+)) 



as in Section 1.2. By Proposition I2.10[ we have only to take an £ G Lo such that 
( , )^|) and ifev are standard. Here we note that 

{u,^)iv = l{ifiv{u),v) - {uJMv))) 

First suppose that cq is even. Then ( , )y* is nondegenerate on f/(o+) by Propo- 
sition 12.91 So there exists an £ G Lo such that ( , )iy is equal to the stan- 
dard alternating form ( , )o on f/(o+)- If Piv = 0, then we have proved that 
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iV = V{0, 0, c+, c_)gygn. Suppose ipiv 7^ 0. Then we can take a nonzero element Vq 
of f/(o+) such that fiv{v) = {vo,v)o for all v G t/(o+)- Let 5* denote the subgroup of 
Lq defined by 

(2.12) S = {geLo\ {gu,gv)o = {u,v)o for all u,v e f/(o+)} = Sp,„(F). 

Then we can take an element £0 of S such that £0^0 = Gc++i- We have 

Since (ec++i, ec++i)o = 5i,co, we have proved io^V = V{0, 0, c+, c_)g^g„. 

Next suppose that cq is odd. Since the bilinear form ( , )y is nondegenerate on 
f/(o+), we can take a nonzero element Vq G ?^(o+) such that v^y(f) = (fo,f)y for all 
f G f/(o+)- Since (fO)^)v = for all v G V9y^(0), it follows that vq ^ Lpy^{0). Take 
an element £ of Lq such that 

£fo = Cc^+ci and that ^(y9y^(0) = W. 

Then we have (ec++ci,f)|y = for all v G f/(o+) and v?^y(ec++i) = 5i,ci. Since ( , )||) 
is nondegenerate on = (^9^(0), we can take an element io of 

L', = {ieLo\ ie,^+c, = ec++ci and iW = W} ^ GL,,,_i(F) 

such that ( , )0y = ( , )o on ly. Thus we have ioiV = V{0, 0, c+, c_)odd- 

(ii) By Proposition I2.10[ we have 

£V = y ^ ( , = ( , )f and = 
for £ G Lo. If y = \/(0, 0, c+, c_)0,en, then ( , )f = ( , )o and cpv = 0. Hence 

£1/ = \/^£g 5^ Sp,^,(F) 

with the subgroup S of Lq defined in fl2.12p . Next suppose that V = V{0, 0, 0+, C-)l^^^ 
Then ( , )y* = ( , )o and V9v(f) = (ec++i,f)o for v G f/(o+). Hence 

iV = V <^=^ £ G S" and ^Cc^+i = Cc^+i. 

Finally suppose that V = 1^(0, 0, c+, c_)odd- Then ( , )y* = ( , )o and ipvicc^^i) = 
6i^ci for i = 1, . . . , Cq. Hence 

£V = V {iu, Cv)'q = {u, f )o for u,v E f^(o+) and (pev = fv i E S' 

where S' = {£ E Lq \ i{ec^+co) = ^c++co, £{W) = W and {£u,iv)Q = {u,v)q for u,v E 
W}^lx Sp,„_i(F). 

(iii) If we fix c+ and c_, then we have [r]„/[r]c+ [r]c_ [r]c„ choices of the pair {V fl 
Uq,V n Un)- On the other hand, every element of M(f/(_,_), f/(_)) is contained in 
the Lo-orbit of = 1/(0, 0, c+, c_)odd, V"(0, 0, c+, C-)"^^.^ or V {0,0, c+,c_)l^^^ as is 
proved in (i). Since \LoV\ = ipcoi^) (ii)' we have the desired formula for HV. □ 

Let Ph be the parabolic subgroup of H = Rn defined by 

PH = {geH\ gU(+) = f/(+) and (?([/(+) © f/(o+)) = f/(+) © f/(o+)} 
= {g H \ gU(+) = f/(+) and gU(-) = f/(_)}. 
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Then the unipotent radical Nh of Ph is written as 
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H 



h 




and 



A 











B 











C 



A G GLe+(F), B e GLeo(F), C e GL, 




is a Levi subgroup of Ph- Let A^(p, g;F) denote the space of p x q matrices with 
entries in F. Then every element of is written as 




n{A,B,C) = h 

with A e M{c+, Co; F), B e M{co, c_; F) and C G M{c+, c_; F). 

Fix a V G M(f/(+), f/(_)). By Definition Ol a full flag : Vi C 
standard if 



C K is called 



Vn = V and Vi = {Vi n f/, 



{+-)) 



(0)y 



for all i 



1, . . . , n. Here we write 



(+) 



Lemma 2.12. (i) For every g G A?"//, there exists a linear map 

/, :Fnf/(o)^f/(+-) 

such that gv = V + fg{v) for allv&Vr\ f/(o) . 

(ii) The map g ^ fg is a surjection of Nh onto the space of ¥ -linear maps 
Hom(\/nf/(o),f/(+-)). 

Proof, (i) For g = n{A, B, C), we have gn+{v) = vr+(t>) + An+{v) for v G V fl f/(o). 
Here the matrix A is naturally identified with the linear map A : t/(o+) — ^ ^(+)- 
Since 

/. 








'CO 



Co 



J 



'CO 





-T ^AT 



we have g7r^{v) = tt^{v) — Jc_ ^BJcg7T^{v). Hence 

gv = gn+{v) + g7i^{v) + gnziv) =v + A']t+{v) - Jc_ *S Jc(,7r_(t;) 

So we can write fg{v) = An+{v) — Jc_ ^BJcgir^^v). 

(ii) For every element / G Hom(V^ fl f/(o), f/(+_)), f{v) is written as 

/(t;) = A7r+(t;)- J,_*54,7r_(t;) 

with some A G A^(c+, Cq; F) and 5 G M{co, c_; F). 



□ 
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Let J" : Vi C ■ ■ • C 14 be a full flag in f2"+^ such that K = V. Define subsets h 
and J2 of / = {1, . . . , n} by 

h = hiJ") = {71, ...,lc} = {^eI\V,n ?7(+_) ^ n f/(+_)}, 
/2 = hiJ") = {Si, ...,6,,} = {tel\V^n = V,-i n f/(+_)} 

where c = c+ + c_ and 71 < ■ ■ ■ < 7c, 5i < • ■ ■ < 5co- Let t(J^) denote the 
permutation 

r(J^) : (1 2 ■ ■ ■ n) K> (71 ■ ■ ■ 7^5^ ■ ■ ■ 
of / and £(r(J-')) the inversion number. Then 

£(r(^)) = (c+ + c_)co- ^ dim(\/,nf/(+_)). 

Proposition 2.13. (i) Let J" : Vi C ■ ■ ■ C 14 6e a full flag m F2"+^ swc/i that 
Vn = V . Then there exists a g & Nh such that gT : gV\ C ■ ■ ■ C gVn is standard. 

(ii) Let : Vi C ■ ■ ■ C Vn and J-'' : V( C ■ ■ ■ C be two standard full flags in 
]p2n+i_ Suppose J-"' = gj-" with some g = gNQi ^ NhLh = Ph. Then J-"' = giJ^. 

(iii) If¥ = ¥r, then \NhJ^\ = 

Proof, (i) Take elements Vi & Vi — for i E I2 and define a subspace = 
0ie72 of V^. Then we have V = U(^+^) © W and 

= (i^i n © (v^, n w) 

for z = 1, . . . , n. 

The subspace is written as = {v + f{y) \ v G F fl f/(o)} with some / G 
Hom(y n f/(o), f/(+~)). By Lemma r2.12[ there exists a (7 G A''// such that gW = 
V n ?7(o). Hence the flag gj^ is standard. 

(ii) Suppose V, = {V, n f/(+_)) © (\/, n f/(o)), = (\// n f/(+_)) © (V;' n f/(o)) and 
gVi = V- with some g = gnQL ^ N^Lh = Ph- For an element v G fl f/(o), write 
(^f = "Wo + '^i with Vq G V/nf/(+^) and G V^/nf/(o). Since giv = g^^gv G f/(+_) + fi, 
it follows that giv = v[. Hence 

gL{v,nu^o)) = v:nU(^o). 

Since giVi fl = fl and since gN acts trivially on f/(+-), it also follows 

that gilVi n f/(+_)) = n f/(+„). 

(iii) We may assume that J-" is standard. Let g be an element of Nh- Then 

= ^ ^ /^(r, n f/(o)) C n f/(+_) for all t G Is. 

Take t>j G (V^int/(o)) — (V^i_i fl f/(o)) for i G /2. Then {t>j | i G I2} is a basis of VnU(^o). 
The above condition is equivalent to 

fg{vi) eV,n for all ieh. 

The number of such fg is r^^^^a '^™*^^'^'^(+-)\ On the other hand, the number of 
elements in Hom(V^ n t/(o), f^(+~)) is r('=++^-)^o. Hence \NhJ^\ = r'^^^^-^)). □ 
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2.5. Proof of Theorem 11.21 Theorem 11.41 Proposition 11.71 and Theorem 

Theorem 11.21 follows from Proposition 12.31 (i), Lemma 12.81 and Proposition 12.111 
(i). Proposition 11.71 is proved in Proposition 12.111 

Proof of Theorem \1.4\ By Theorem 1 1.2 1 we may assume that t = {Uq, Ud, V) {d = n— 
a-b) with V = V{a, b, c+, c_)odd, V = V{a, b, c+, C-f^^^^ or V = V{a, b, c+, C-)l^^^. 
Since 

(2.13) \Gt\ = \{{gUo,gUd) \ g G G}\\RdV\ = \M\\PUd\\RdVl 

we have only to compute \PUd\ and |-Rd^|- By (12.81) . we have 



(2.14) \PUd\ = r 



d{d+l)/2 F| 



[r]d[r]n-d 



On the other hand, it follows from Proposition 12.31 (ii) and (iii) that 

(2.15) \RdV\ = |Q(i?,nL^J\/| =r«'^-")("-'^+i)-'^('^+^))/2i%^ 

[na[r]b 

By Proposition 12. Ill (iii). we have 

(2.16) |(«,nL.J^|^^j-|ik|^*iM. 

So the assertion follows from fl27[3l) . ( KWf . (127[5|) and f l216|l . □ 

Let Ud and V be as above. Write H = Lw^ n Rd = GLrf(F). Then Rd = P n Pu^ 
is decomposed as Rd = QH. By Proposition 12.51 (ii). 

R{t) = pnPu,nPv = RdnPv = Qv{H n Py) 

where Qv = Q H Py- Define a parabolic subgroup 

PH = {9eH\ gU(^+) = [/(+), = f/(_)} 

= {geH\ gUi+) = (?([/(+) © f/(o+)) = f/(+) © t/(o+)} 

of H. Then we have a Levi decomposition P^ = NhL where L is defined in Section 
1.3. Since Nh C H n Py C Ph, we can write H n Py = NhW with Ly = L n Py. 
Thus we have: 

Lemma 2.14. We have a bijection 

Qy X Nh X Ly 3 {g, h, £) ^ gM G R{t) =PnPu^n Py. 



Proof of Theorem {TE (i) Let : C ■ ■ ■ C K be a full fiag in Mo{V). By 
Proposition 12.51 (i). there exists a G Qy such that 

gVi = {gV, n Wo) © {gV^ n W^) © {gV, n W2), 

gVinWo = ¥ei®---® Fe^^cj-) and that gVi nW2 = Fe„+d+2 © ■ ■ ■ © ¥en+d+i+b,{T) 
for i = 1, . . . , n. 



28 



TOSHIHIKO MATSUKI 



Consider the full flag gVx^ n W^i C ■ ■ ■ C gVx^ n Wi in Wi. By Proposition 
(i), there exists an /i G such that 

hgVi. nWi = {hgV, n © f/(_))) © {hgV H f/(o)). 

for i = 1, . . . ,n. So the flag hgj^ is standard. 

(ii) Let J-" and J-"' be two standard full flags in Mq{V) such that gT = T' for some 
g G -R(t). By Lemma [2.14[ we can write g = gqgNgL with (jfg G Qy, fi'Af ^ A?"// and 
gi G Ly. Since gNgh ^ -Z^Wi, it follows from Proposition 12.51 (ii) that g^giJ^ = J^'- 
By Proposition 12. 131 (ii), we also have giJ^ = T' as desired. 

(iii) Let J-" : Vi C ■ ■ • C be a standard full flag in M^iy). Considering the full 
flag n W\ C ■ ■ • C V\d n W\ in VTi, we define a permutation 

r' = r'(^):(Ai---Ad)h^(7i---7c5i---5co) 

of the subset {Ai, . . . , A^} in {!,..., n). Let £(r') be the inversion number £(r') = 
|{(7i, 5j) I 7i > Then we have 1{t) = £{a) + £{t') where 

a : (1 2 ■ ■ ■ n) (ai ■ ■ ■ aaK ■ ■ ■ Ad/3i ■ ■ ■ /3fe) 
and r : (1 2 ■ ■ ■ ra) t-^ (tti ■ ■ ■ a^Ti ■ ■ ■ Ich ■ ■ ■ ■ ■ ■ l^b) 

are as in Section 2.4 and Section 1.3. 

By Proposition 12.131 (ii) and (iii), we have \NHLyJ^\ = r^^"^ ^\LvJ-'\. On the other 
hand, by Proposition [23] (ii) and (iii), we have \QvNhLvJ-'\ = [r]a[r]br^^°'^\NHLvJ-'\- 
Hence \R{t)J^\ = [r]a[r]y^^^\LvJ^\ by Lemma □ 

3. Orbits on GL„(F)/fi 

3.1. Preliminaries. Let F be an arbitrary field. Let 

1^1 C ^2 C ■ ■ ■ C K-i C F" and Wi C W2 C ■ ■ ■ C Wn-i C F'^ 

be two full flags in F'^. (Write Vq = = {0} and Vn = = F".) Define 
(iij = dim(\/i n Wj) for z, j = 0, 1, 2, . . . , n and = rfij - rfj.ij - djj.i + cii_ij_i 
for z,j = 1,2, . . . ,n. 

Proposition 3.1. The n x n matrix {cjj} is a permutation matrix. 

Proof. Since Vi.iHWj.i = {Vi.ir]Wj)r]{VinWj_i) and since VinWj D {Vi.inWj) + 
(Vi n Wj^i), we have > 0. On the other hand, ^"^j^ Cij = dnj — doj — dnj-i + 
rfoj-i = J - (j - 1) = 1 and X]j=i = di^n - di^o - c?j-i,n + c?i,o = i - {i - 1) = 1. 
Hence {cij} is a permutation matrix. □ 

We also have the following by the same arguments as above. 

Proposition 3.2. The following four conditions are equivalent: 

(i) Cij = 1. 

(ii) dij — 1 = di-ij = c?jj_i = 

(iii) Vi n Wj g Vi.i n Wj = n = v^i n 

(iv) Vi n ly,- 5 (\/i_i n Wj) + {v n Wj.i). 
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Remark 3.3. (Bruhat decomposition of G = GL„(F)) Let Vi C V2 C ■ • ■ C Vn-i be 
the canonical full flag defined by = Fei © ■ • • © Fcj for i = 1, . . . , n — 1 with the 
canonical basis ei, . . . , of F". The subgroup B of G defined by 

B = {g E G \ gVi = for i = 1, . . . , n — 1} = {upper triangular matrices in G} 

is a Borel subgroup of G. 

Let Wi C W2 C • ■ ■ C Wn~i be an arbitrary full flag in F". By Proposition 13. we 
can define a permutation i = of {1,2,..., n} determined hj i = <^==^ Cij = 
1. Suppose i = Then by Proposition 13. 2[ we can take vectors Vi E ViH Wj for 
i = 1, . . . ,n (for j = 1, . . . ,n) such that 

It follows that Wi, . . . , f„ is a basis of F" such that Vi = Ft>i©- ■ -©Ffj for i = 1, . . . , n. 
It also follows that f . . . , fi{n) is a basis of F"' such that Wj = Ft>j(i) © ■ ■ ■ © ¥vi(^j) 
for j = 1, . . . , n. Define n x n matrices 

g = {viV2 ■■■Vn) and w = {qj} = (6^(1)64(2) ■ ■ ■ ei(„)). 

Then g E B and = W^j for j = 1, . . . , n. Thus we have proved G = Uwgw -^"^-^ 
where W is the subgroup of G consisting of all the permutation matrices. 

3.2. Sp2„-orbits on GL2n/-B- Let ( , ) denote the alternating form on F^" defined 
by 

/ \ _ J ^i,2n+i-i for i < n, 
[ -()j,2n+i-j for I > n. 

Define a subgroup H = {g E G \ {gu,gv) = {u,v) for all u,v E F^"} = Sp2„(F) 
of G = GL2n(F). Let C 1^2 C ■ ■ ■ C V2n~i be a full flag in F^". Then there 
corresponds another ("decreasing") full flag V^-*- D V^^ D ■ ■ ■ D V^_i in F^" defined 
by 

V--^ = {uE F^" I (m, v) = for all v E Vt}. 

Define dij = dim(Vi fl Vj^) and = (ijj-i — dij — di^ij^i + di-ij. Then we have 
the following two propositions by Proposition 13.11 and Proposition 13.21 

Proposition 3.4. {cij}f^^i is a permutation matrix. 

Proposition 3.5. The following four conditions are equivalent: 

(i) Cij = 1. 

(ii) dij-i — 1 = dij = = di-ij. 

(iii) V, n y^^i ^v,n \// = n Vj^_, = v,^^ n v^^. 

(iv) V, n v^^i ^ (F, n v^^) + (F,_i n i^^J 

Since the orthogonal space of Vjfl V^-^ is V^-^ + Vj, we have dim(V^-'- + V,) = 2n — dij. 
So we have 

rfj- i = dim(\/j n Vi^) = dim V;,- + dim - dim{Vj + V^^) 
= j + {2n -i) - {2n - dij) = dij 
and hence Cjj = Cj^i. 
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Lemma 3.6. Suppose that Cij = 1. Then we have {u,v) ^ for all u E (Vi H 

v^^L,) - n and V e (v, n v^,) - (v,-_i n v^,). 

Proof. Suppose ue{V,n V^di) - n r^di) and v G {Vj f] V^^^) - {Vj^i f] V^^). 
Then 

{u, Vj-i) = {0} and Vj = Vj-i © ¥v. 

If {u, v) = 0, then 

{u, V,) = {u, Vj.i © ¥v) = {u, = {0} 

and hence u E ViH = fl Vj^-^^ by Proposition 13.51 (iii). But this contradicts 
the choice of m. □ 

Proof of Proposition \1.9[ Suppose Cj_j = 1. Then we can take an element v G 
(K- nK-*-^) — n V^-'-;^) by Proposition 13.51 (iii). By Lemma 13. 6 j we have {v,v) 7^0. 
But this contradicts that ( , ) is alternating. □ 



Proof of Proposition lTJU (i) We will prove this by induction on n. Take a pair 
such that i < j and that Cjj = 1. By Lemma we can take a Vi E ViH V^-^^^ and a 
Vj E VjCi V^-^i such that (f j, Vj) = 1. Put U = ¥vi © ¥vj. Then we have a direct sum 
decomposition F^" = ?7 © JJ-^. li k < i - 1, then 14 C U^. Hi < k < j - 1, then 

14 = ¥v, © (v^ n u^) = {Vk nu)(B (v, n u^). 

lik>j, then D t/. Hence for every A; = 0, . . . , 2n, we have Vk = {Vk n f/) © (V^ fl 
f/^). We also have V^/ = (K/ n f/) © (V/ n f/^) and 

Vk n = (14 n V/- nu)® {Vk n n f/^) 

for A;, £ = 0, ... , 2n. 

We can consider the full flag Vi fl f/-*- C ■ ■ ■ C V2n-i H f/-*- in f/-*- neglecting 
the two coincidences n If-^ = (1 JJ-^ and fl f/-^ = V^- n f/-^. Define 
d'k^^ = dim Vfcn?7^nf/-^, d'/,^^ = dim Vfent/^nt/, c'^^^ = d'^^^_^—d'^^ (^ — d'^_^ i^_^+d'k_i^i and 
4/ = + for /c, £ G / = {1, . . . , 2n}. Then 4,^ = d'^^^+d^^^ 

and Cfc,^ = £ + c'^ ^. For A; G / — {i, j}, we see that Vk-i r\U = Vk^U . So we have 
Cfc,£ = ^'^^ Cfc/ = 4 £ for fc, £ G / — {i,j}. 

By the assumption of induction, we can take a basis 

f 1, . . . , . . . , Vj-i, Vj+i, ...,V2n 

of [/-*- such that 

VknU^= Ft;, 
ee{i,...,k}-{i,j} 

and that (ffc, f,) = Ck/ for A;, £ G / — {i, j}- Thus the basis f 1, . . . , f2n of F^" satisfies 
the desired properties. (Remark: We may take (z, j) = (ii, ji) = (1, ji) in the above 
proof. But we will need such a general argument as above in the proof of Proposition 
[TH) 
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(ii) Since F = consists of r elements, we have r — 1 choices of Vi in Vi = 
Vi n V^|"_i- Fix Vi. Then we have r^^~^ choices of fj^-i in Vj-^ = Vj^ fl V^-^ such that 
(f 1, f j^) = 1. Write 

£2 = \{i\2<i and a{2) > 

Then we have ji = a{2) = £2 + 2. 

Fix f 1 and Vj-^ . Then the subspaces U = ¥vi © Ff and f/-*" in (i) are determined. 
Next we take Vi^ E Vi^H = F. So we have r — 1 choices of Vi^. Fix fjj. We see 
that 

j2 - 2 if ji < js, 



dim(\/j2 n [/ 



j2 - 1 if ji > J2- 



On the other hand, if we write 

£4 = |{i I 4 < i and cr(4) > 

then we have 

32 - 4 if ji < js, 
j2 - 3 if ji > J2- 

Hence we have r^*+^ choices of Vj^ E Vj^ fl U-^ such that {vi^^Vj^) = 1. 

Repeating this procedure, we have r — 1 choices of Vi^, and r^^^+i choices of Vj^ if 
we fix vi,Vj^,..., Here 

^2fc = \{i \ 2k < i and o-(2A;) > o-(z)}| 

for /c = 1, . . . , n. Since £(0") = ^2 + ^4 + ■ ■ ■ + ^2n) "we have (r — i^^^-^+^C'^) choices of 
the bases Vi, . . . , V2n- D 

3.3. (52n-orbits on GL2n/-B- Let H = Sp2„(F) be as in the previous subsection. 
Let Q2n denote the subgroup of H defined by Q2n = {q ^ H \ ge2n = ^2n\- Then 
Q2n stabihzes the hyperplane W = ¥e2® ■ ■ ■ ® Fe2n = (IFe2n)"^ in F^*^. 

Let Vi d ■ ■ ■ C. V2„-i be an arbitrary full flag in F^". Let S denote the subset of 
J X J defined hj S = j) | Vi n l/-^^ ^ W}. Define 

5o = ESWnV^^dW and v,.^ n V^'l^ C W]. 

Lemma 3.7. Suppose G Sq. Then we have: 

(i) Cij = 1. 

(ii) Vi n Vj^_, nw = V-i n v^^_, = Vin = v^i n r/. 



Proof, (i) If (i, j) G So, then (1^^ n \//) + {V^i n l^^-^J C W. So we have (^^ n \//) + 
n Vj^j) ^ViH Vj^^ and hence Cij = 1 by Proposition 13.51 
(ii) By (i), it follows from Proposition 13.51 that 

V n Vj^_, ^ n v,.^, = n = v., n v,.^. 

So the assertion is clear. □ 
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It is clear that 

(3.1) {^,J) G 5 and z < z', j > f =^ (z', j') G S. 

So the subset 5*0 determines S by 

(i, j) G S" 2 > io and j < jo for some (io, Jo) e Sq. 
For example, if n = 2 and 5*0 = {(2, 1), (4, 3)}, then 

5 = {(2,1), (3,1), (4,1), (4, 2), (4, 3)}. 

Proof of Proposition \l.l^ By Lemma [XTJ we may assume that Xi < X2 < ■ ■ ■ < Xg. 
If i < j and yj < i/i, then it follows from {xi, Ui) G Sq that (xj, + 1) G 5 by (13.11) . 
But this contradicts that {xj,yj) G 5*0. Thus we have 

yi<---<ys- 

By Lemma 13. 7^ the numbers Xi, . . . , x^, j/i, . . . , z/^ are distinct. □ 

Write /(^) = / - {xi, . . . , x.,, yi, . . . , 

Proof of Proposition \1.13[ (i) We will prove this by induction on n. First assume 
that s < n. Then we can take a pair («, j) in J(^) such that i < j and that Cij = 1. 
By Lemma [3l6| we can take a f j G VinVj^i and a f ^ G Vjfl Vji*:]^ such that (fj, Vj) = 1. 
S, then 

On the other hand, if (i, j) G S", then there exists an (x^ , yt) G such that i > Xt and 
j < yt- By Lemma 1221 there exists a v E Vx^H Vy^-i such that v ^ ly. Since Xt < z, 
we have {v, vf) = 0. If Vi ^ VT, then we can replace Vi by Vi + av E Vi H Vj^-^ H 
with some a G since Vxt cVi and V^-^ C V^":^!. Thus we may assume that Vi G W. 
In the same way, we may also assume that vj G W. 

Put f/ = ¥vi © Ff j and consider the direct sum decomposition F^" = U (B as 
in the proof of Proposition ll.lOl Since U C W, is contained in U^. So we may 
assume that we have chosen a basis 

Vi,.. fi_i, Vi+i, . . . , Vj-i, Vj+i, . . . ,V2n 

of satisfying the conditions (a), (b) and (c) for JJ-^ by the assumption of induc- 
tion. Then the basis vi, . . . , V2n of F^" is a desired one. 

Finally we consider the case of s = n. By Proposition I1.10[ we can take a basis 
fi, . . . , V2n of F^" such that Vi = Ft>i©- ■ -©Fvi for z = 1, . . . , 2n and that {vi, vj) = Cij 
for i < j. By Lemma [3.7[ f^^j, . . . , Vx^ are not contained in W. So we can normalize 
them so that 

{Vxi,e2n) = ■■■ = {Vx„e2n) = L 

We can also normalize , . . . , Vy^ so that {vxt ,Vy^) = St where 

_ 1 1 if Xt < yt, 
* I -1 if Xt > yt. 



ORTHOGONAL TRIPLE FLAG VARIETY 33 

If yt < Xt, then {yt,Xt) ^ 5*. So we have Vy^ G W. On the other hand, if yt > Xt and 
% ^ W, then we can replace Vy^ by Vy^ + av^t £ W with some a G F^. Thus the 
basis Wi, . . . , f 2n of F^" satisfies the desired properties. 

(ii) By Proposition I1.10[ we have (r — \^'^r'^+^('^) choices of the bases without the 
condition (c). By the first condition 

Vi &W ioi i ^ xi, . . . ,Xs 

in (c), the number of choices of Vi becomes 1/r of the number without the condition 
(c) for each i such that 

(3.2) (z,j)^5-5owithc,,, = l 

as in the proof of (i). Since there are m = m{J-') indices i satisfying the condition 
(13. 2p by the definition, we divide the number by r™^. On the other hand, the second 
condition 

in (c) is the condition on the length of vectors Vx^, ■ ■ ■ ,Vx^. So we divide the number 
by (r — 1)'^. Thus we have (r — i^^-^f^'+^i'^)-"^ choices of the bases. □ 

Proof of Theorem \1.14\ (i) For each partition / = /(^) LJ/(x) U J(y) and each {qj} G 
C(/(^)), we construct a "standard" basis Ui, . . . ,U2n of F^" as follows. We can take 
a unique subsequence ii < ■ ■ ■ < ins in I{A) such that Cj^j^ = . . . = Cj„_^j-„_^ = 1 
with some ji, ■ ■ ■ , jn-s ^ I{A) and that it < jt for t = 1, ... ,n — s. Define 

Ui^ = Cg-i-i, . . . , Uj^_^ = Cn, 
'^jl ^2n—si ■ ■ ■ 1 '^jn-s ^n+l; 

Mzi = ei + 62, =61+63, . . . , Ux,_^ = 61 + 6s, U^, = 61, 

Uyi = £ie2n-l, Uy2 = ^2G2n-2) • • • , Uy^_^ = S s-ie2n-s+l, 

Uy, = Ss{e2n — G2n-l — ■ • ■ — 62„_s+l)- 

Then the basis vectors mi, . . . , U2n satisfy the properties: 

(3.3) (wj, Uj) = Cij for i < j, 

(3.4) Ui E W for i Xi, ... ,Xs 
and 

(3.5) {Ux^,e2n) = ■■ ■ = {Ux,, 62„,) = 1. 

Let vi, . . . , V2n be the basis of F^" given in Proposition 11.131 (i). Let g be the 
element of GL2n(F) defined by gvi = Ui for i = 1, ... ,2n. By (13. 3p and Proposition 
051 (i) (b), g is an element of H = Sp2„(F). By ([231), (IS3D and Proposition [HSl 
(i) (c), we have 

W = ¥{ux,-Ux,)®---®¥{ux,_^-UxJ® ¥ui 

i^{xi,...,Xs} 

= ¥{vx,-Vx,)®---®¥{v,^_,-VxJ® ¥vi. 

i({xi,...,Xs} 
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So the element g stabilizes W. Hence we have ge2n = /3e2„ with some (3 E ¥^ . 
Moreover we have 

Thus we have proved g G Q2n- Since the flag gVi C ■ ■ • C gVn is written as 
gVi = ¥ui © ■ ■ ■ © ¥ui {i = 1, . . . , 2n — 1) using the standard basis Ui, . . . , Un, we 
have proved (i). 

(ii) is clear and (iii) follows from Proposition 11.131 (ii). □ 

3.4. Proof of Theorem 11.191 Proof of Lemma \l.l^ Let ^{2k, s) be the number 
of the words with 2k letters consisting of 

a,b, . . . , A,B, . . . ,X,Y 

with \Ix\ = \Iy\ = s. There are k — s pairs of a,b, . . . , or A,B, . . . and we can assign 
a signature +1 or —1 to each pair according as they are small or capital. So we have 



i{2k, s) = 2'''\2k - 2s - 1)(2A; - 2s - 3) • • ■ 1- 



(s!)2(2A;-2s)! {s\Y{k - s)V 

Thus we have the desired formula for ^{2k) = ^^=o^(2^> We can prove C,{2k — 
1) = ^^1(2^^ - l)!/s!(s - 1)!(A; - s)! in the same way. □ 

Proof of Theorem \1.1S\ (i) We can divide every word w into the subword Wi con- 
sisting of a, (3, +, — and the subword W2 consisting of the other letters. There are 
^n-k Qjioices of Wi and C,{k) choices of W2 if the length of W2 is k. Considering the 
number of partitions of w into two subwords with n — k letters and k letters, we get 
the desired formula 



|AG\M xMxMo\ = X^4"-^(^') 



k=0 

The proof of (ii) is similar because we have only to consider words without a and 
(3. □ 

3.5. Proof of Theorem 11.201 Proof of Theorem \1.2(A (i) is proved easily. 

(ii) By the same arguments as in the proof of Proposition [2]3] (iii), we have | {N-^y^ n 

Ql'^y]^ = ^bd+b{b-l)/2 ^ ^{{n-a){n-a-l)-d{d-l))/2 g^^^ hcUCC 

(3.6) [(gnGOl^l =r«"-")("-»-i)-'^(^-i))/2i!tf 

[r]a[r]b 

Noting that Rd H Lwi C G', we have 

\{RdnG')v\ = \(QnG'){RdnLw,)v\ = r^'^--^^^"-^-'^-''^'^^^^^^ 

On the other hand, we also have \{P n G')Ud\ = r'^^'^~'^^^^[r]n/{[r]d[r]n-d) by fIXB]) . 
Combining with f l2.16p . we get the desired formula 

\G't\ = \M'\\iPnG')UMRdnG')v\ = |m°| ^7;°^ \ □ 
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4. Appendix 

Let F be an arbitrary field and let F" = © [/_ be the direct sum decomposition 
of F" with f/+ = Fei © ■ ■ ■ © ¥em+ and f/_ = Fe^^+i © ■ ■ ■ © Fe„ (n = m+ + m_). 
Let H denote the subgroup of G = GL„(F) defined by 

H = {geG\gU+ = U+, gU^ = U^}. 

In this appendix, we will give a proof of the if-orbit decomposition on the full 
flag variety of G = GL„(F). 

Let 7r+ : F" — 7- and 7r_ : F" — )■ f/„ denote the projections with respect to the 
direct sum decomposition F" = © For a full flag J-" : Vi C ■ ■ • C in F*^, 
deflne 

= dim(7r+(V,) n V,-), = dim(7r_(V;-) n V^) 

for i, j = 0, . . . ,n and 

— rl"^ — rl"^ — d'^ -I- (7^ r- ■ — r+ -I- r~ 

for i, j = 1, . . . ,n. 

Lemma 4.1. (i) {cij} is a permutation matrix. 

(ii) i > j =^ cfj = and i < j =^ c~j = 0. 

(iii) dj = Cj^i. 

Proof, (i) As in the proof of Proposition 13. 1[ we have cfj > and c~j > 0. On the 
other hand, we have 

n 

= din - dtfl + - dt-ifl = dim7r+(\/i) - dim 7r+(\/i_i) 

i=i 

n 

and = di,n - dlfl + c^r-i,n - ^i"-i,o = dim(yi n - dim(\/i_i n 

i=i 

Hence Qj- = dim Vi— dim Vi-i = 1. In the same way, we also have J2'i=i ^ij = 1- 
So the matrix {qj} is a permutation matrix. 

(ii) If i > j, then rc+iVi^i) fl V,- = 7r+(\/j_i) fl Vj 011+ = Vj fl U+. In the same way, 
we have ir+iVi) nVj = Vj H f/+. Hence c^j = 0. The second formula is similar. 

(iii) Suppose i < j. Then it follows from (ii) that 

Cij = <j = - dim(7r+(Vi) + Vj) + dim(7r+(r,) + Vj^i) 

+ dim(7r+(\/,_i) + V,) - dim(7r+(V^,_i) + V,^,) 

since dfj = dim7r_|_(Vi) +dim V^- — dim(7r+(Vi) + 1^) fori,j = 0,...,n. It also follows 
from (ii) that 

= = - dim(7r_(\/,) + V,) + dim(7r_(\/,) + V;_i) 

+ dim(7r_(y,„i) + Vj) - dim(7r_(F,_i) + V;_i). 

Since 7r+(Vfc) + Vi = 7r_(Vfc) + for A; < £, we have = c^^j. □ 
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Define subsets 

/(+) = {kf, k+^_,,} = {tel\ c+ = 1}, 

A-) = {^i"' ■ • ■ ' Kn—s} = e / I C'i = 1}, 
-^(1) = {^1) ■ ■ ■ , is} = {?' G / I Cij = 1 for some j > i}, 
1(2) = {ji, ■■■Js} = {j^I\ Cij = 1 for some i < j} 
of 7 = {1, . . . , n} with kf < ■ ■ ■ < k^^_^, k^ < ■ ■ ■ < k~__^, ji < • • • < js and 

Cit,jt = 1 for i = 1, . . . , s. 

Then I = /(+) U /(_) U /(i) U /(a). Write /(+) U /(_) = {ki, . . . , kn-2s} with ki < ■ ■ ■ < 
kn-2s- Define a permutation 

a : {12- ■ - n) ^ {is - ■ ■ hki ■ ■ ■ kn-2sji ■ ■ -js) 

of / and the inversion number i{a). (Remark: Let r denote the permutaion corre- 
sponding to the matrix {qj}. Then we can prove £{t) — s{n — s) — 2i{a).) 

Proposition 4.2. For every full flag J-" : Vi C • • • C Ki-i in F", define Cij — 
ct-i + cJ-i above. 

(i) We can take a basis Vi, . . . ,Vn ofW^ such that 

(a) Vi = ¥vi © • ■ ■ © Ft',, for i = 1, . . . , n. 

(b) Cj^j = 1 =^ Vi e U+, Cj^j = 1 =^ Vi G f/- and 

(ii) Define a basis Ui, . . . ,Un of¥"' by 





= et fort^l,. 


. .,m+-s, 




= e^++t /or i = 


1, . . . , m_ — s, 






fort^l,...,s 




= e^_^_s+t /or i 





anc? define g G G by gvi = Ui for i = 1, . . . ,n. Then g E H. 

(iii) //F = F^, then the number M{J-) of the bases satisfying the conditions in (i) 

is 

(j- — 2'\»^-Sj.(('n+-s)('»+-s-l)+(™--s)('n--«-l))/2+^(a-) 



Proof, (i) Suppose c^^- = 1. Then we can take a, v E 7r+(V^) fl T^- such that v ^ 

7r+(l^i) n Vj^i = 7i+{Vi_i) n Vj. If i = j, then Vi = v E U+ satisfies V = V^i © ¥vi. 
Suppose i < j and take a G so that 7r+(fj) = f . Then = Vi_i © Ft'j and 
Vi ^ ?7_. If i;, G then w = G C V^_i a contradiction. Hence ^ U^. Take 
Vj ^ 7r+{vi). Then V^- = T^_i © Ft;^. 

If c^^ = 1, then we take a G 7r_(Vi) fl Vi such that ^ '^-{Vi) ^ V^_i = 
7r_(\^i_'i) n v. Then G C/_ and = y^-i © ¥vi. 

(ii) is clear from (i). 



ORTHOGONAL TRIPLE FLAG VARIETY 37 

(iii) We will prove this by induction on n. First suppose c+„ = 1. Then Vn-i = 
(Vn-i n © f/_ and so the number of the bases vi, . . . , of Vn-i satisfying the 
conditions in (i) is 

_ l^n-s-l^{{m+-s-l){m+-s~2)+{m.-s){m.-s-l))/2+e{a') 

by the assumption of induction. Here a' is the permutation 

a' : (12---n- 1) h-> (is ■ ■ ■ hh ■ ■ ■ kn-^2s~iji ■ ■ ■ js)- 
For each basis t>i, . . . , Vn-i of Ki-i, we have (r — l)r'"+~"^ choices of f„ G f/+ — (f/+ fl 
K-i)- Since 

(771+ — s)(m+ — s — l)/2 = (m+ — s — l)(?Ti+ — s — 2)/2 + (m+ — s — 1) 
and since £{a) = i{<j') + s, we have 

_yy(jr^ = (^r — l'jn-s^{{m+-s){m+-s-l)+{m--s){m-.-s-l))/2+e{a) ^ 

The case of c~„ = 1 is similar. 

So we may assume Cp,„ = = 1 with some p < n. Consider the sub- 
space W = (Vn^i n f/+) © (K_i n of F". Then the number of the bases 
f 1, . . . , fp_i, fp+i, • • • , fn-i of W satisfying the conditions in (i) is 

(^r - l)«-«-l^(("i+-s)(m+-s-l)+(»n_-s)(r7i_-s-l))/2+£(<7') 

by the assumption of induction where a' is the permutation 

a' : (1 2 • • -p - + 1 ■ ■ - n - 1) h^. ■ --iiki ■ ■ ■ kn~2sji ■ ■■js-i) 

of {1 . . . ,p — l,p + 1, . . . , n — 1}. (Note that p = is and n = jg.) For each basis 
Vi, . . . ,fp_i,fp+i, . . . ,Vn-i of W, we have (r — l)r^~^ choices of tip G — V^_i. 
Since £(cr) = ^(o"') + p — 1, we have the desired formula for A/'(J-') noting that 

By this proposition, we can express orbits by "H — ab-symbols" as in Fig.l (c.f. 
jMO90] ). We can easily count the number of orbits: 

min(r?i+,m_) 

TT TTT TT- 

2^s!(m+ - s)!(m_ - s)! 

Corollary 4.4. For eac/i full flag T in F", define s and a as above. If¥ = ¥r, 
then 

\HJ^\ = (r - l)V^("-^-i)-^('^)[r]^^[r]^_. 

Proof. Note that 

\H\ = (r"^+ - l)(r™+ - r) ■ ■ ■ (r"^+ - r™+-i)(r"^- - l)(r"^- - r) ■ ■ ■ (r™- - r™— ^) 
= (r - l)"r('"+('"+-^)+'"-('"— 

Since (m+(m+— l)-|-r/i_(m_ — 1))/2— ((rri_|_— s)(m_|_— s— l)-|-(rri_— s)(r/i_— s— 1))/2 = 
s{n — s — 1), we have 

\HJ^\ = m/ATiJ') = (r - l)V^("-^-i)-^(-)[r]™^[r]™_. 

□ 
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